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Head-Mounted Displays 


e Used for VR 
e IMU for position and orientation tracking 


e Display with optics for view 


a 
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HMD Display Process 


e Application generates left/right eye 
buffers 


e VR Compositor constructs unified frame 
buffer 


- Inverts distortion caused by headset optics 


- Inverts pixel intensity offsets in individual 
Left Eye Buffer 


Optics Transform Adjust Pixel Intensity Present Image 
Right Eye Buffer 
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HMD Requirements 


e Hard real-time 
- New frame 90 times per second 
- Bounded latency requirement 
- Vary amount of work to hit target 
e Definitely not part of the desktop 
- No window management 
- No “regular” apps 
- Just showing a static image is “hard” 
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HMD Display Options 


e ICCCM conventions for hiding displays 
+ no kernel or X changes required 
- Involve the desktop. All of them. 
- Does not address latency issue. 
e RandR changes to hide displays 
+ No kernel changes required 
- X changes required 
- Does not address latency issue. 
e Meta display server to manage outputs 
+ No new X protocol 
+ No kernel changes 
- X changes required 
- Does not address latency issue 
e Kernel changes to let apps “borrow” display 
+ Address latency concerns 
+ No visible changes to desktop 
- Kernel changes 
- X changes 
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Get X out of the picture 


e X isn't helping here 
- Adds latency when flipping 
- Even worse when a compositor is running 
e Make HMD app display directly to the device 
- It's already got an FD pointing at the device 
- “just” needs mode setting access. 
e Construct a mechanism to pull display out of X 


and hand to HMD app. 
- Want to leave X able to recover when the app crashes 
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Leases 


e A “lease” is a contract between X and the VR app 
naming a set of display resources 


e The “lessor” (e.g. X server) is the current 
controller of the leased resources and promises to 
leave the resources alone while the lease is active 


e The “lessee” (e.g. VR app) is free to set modes, 
flip frame buffers, DPMS on/off at will. 


e When the lesse terminates the lease, the lessor 
takes the display back. 
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Side Issue - vblank API 


e Current vblank API is a bit of a mess 
- Three functions, one IOCTL 
- Only supports 32-bit frame counter. Wraps in only a few years 
- Only supports microsecond resolution 


e Add two new IOCTLS 


- CRTC_GET SEQUENCE 
e Get last vblank sequence and time 


- CRTC QUEUE SEQUENCE 
e Queue event for delivery at specified sequence 


- 64-bit frame counter 
- Nanosecond resolution (ktime_t internally) 
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Building a Kernel Patch Story 


e It doesn't matter how the code was written 


- People don't usually write books from front to back, why should we expect 
code to be written this way? 


e Patches should tell a consistent story 


- Reviewers time is precious; patches should do one thing which ts 
described in the first line of the commit message 


e Each patch should be review-able by itself 


- Even if the patch only makes sense in context, it should still be possible to 
see the commit message and ensure that the patch does exactly that. 


e Each patch should compile and run by itself 
- Event if it doesn't do anything useful. 


- Without this, you break bisect, which makes debugging in the future much 
(much) harder. 
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Add file parameter to mode 
resource lookup 


e Significant API churn, no functional 
change. 


e Allows adding access control in future 


e Largely mechanical change, easy to 
review 


Ge [VALVE] 


Let render nodes query mode 
objects 


e Permits Vulkan client discovery of 
resources in AcquireXlibDisplay mode 


e Drivers with split render/display files 
would need access to some kind of read- 
only version of the display object. 


e Just changes access control flags 


oe VALVE] 


Lease infrastructure 


e The heart of the change; touches existing 
code as little as possible 


e Each master has list of lessees 
e Each lessee has list of mode resources 
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Lease access control 


e Only for connector, encoder and crtc 
resources 


e Separated from lease infrastructure 
patch, touches only the access control 
code 
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New lease IOCTLS 


e Hooks up the lease infrastructure to user 
mode. 
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Leasing in Action 


DRM Master 


DRM Lease 


HDMI monitor Head Mounted Display 
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X and Leasing 


e X Masks leased objects from X clients 
- Leased outputs always appear to be disconnected 
- Leased CRTCs cannot be used with any output 

e X clients do not expect RandR resources to come 
and go on the fly 
- Even though RandR doesn't require they be persistent, 
- Clients now depend on that. Unintentional ABI. 

e KMS has connectors and encoders, X has only 
outputs. X leasing automatically assigns encoders 
for each output. 
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RandR Additions 


e RRCreateLease 
- Takes list of X outputs and X crtcs 


- Returns XID for lease and FD reference to DRM 
device 


e RRFreeLease 

- Free XID 

- Optionally revoke active lease 
e RRLeaseNotifyEvent 


- Notifies of new and terminated leases 
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Various X fixups 


e Turn off cursor when disabling CRTC 


- Otherwise, the lessee will end up with a 
random cursor on the screen 


e Add a CONNECTOR ID property to each 
RandR output 


- Lets Vulkan driver map between X and KMS 
resources 
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Original Plan 


List X Resources 


List Kernel Resources 
Select Resources 
X Lease Creation 


Kernel Lease Creation 


Start VK 
VkInstanceCreate 


vkQueuePresentKHR 


Subsequent Frames Mode Set 


vkQueuePresentKHR 
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Page Flip 


Vulkan Bits for original plan 


e VK _KEITHP kms display 
- New instance extension 
- Adds structure to pass device fd into 
VkCreatelnstance 
- Drivers can use it instead of their own fd 


e Implement VK_KHR_ display and 
VK_EXT_direct_mode display extensions 
- Create surfaces and swapchains on physical 
devices 
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vkAcquireXlibDisplayEXT 


Start VK 


VkCreatelnstance 
List Vulkan Resources 


Discovery 


List X Resources List Kernel Resources 


Select Resources 


vkAcquireXlibDisplayEXT 
X Lease Creation 


Kernel Lease Creation 


vkQueuePresentKHR 


Subsequent Frames 


vkQueuePresentKHR 
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Page Flip 


Additions for 
VK EXT acquire xlib display 


e Instance already has render node fd 
opened 


e Must create X lease to get master fd 
e Code must now deal with two fds 


e Map RandR output to KMS connector 
using CONNECTOR ID property 
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VK EXT display control 


e Needed for vblank fences on direct mode 
displays 


e Uses new kernel IOCTLs to provide 64-bit 
sequences and nano-second resolution times. 


e Adds new fence types inside drivers, so now 
we have WINSYS (original) and WSI (new). 


e Also adds DPMS support for direct displays 
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Remaining Issues 


e Hide HMD from apps even when no VR 
app Is running. 
- Even fbdev. 


- Maybe some kind of EDID registry that the 
kernel knows about? 


e Wire up hotplug events for 
VK_EXT_display_control 


- Part of the spec, but | haven't needed it yet. 
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Thanks! 


e git://people.freedesktop.org/~keithp/linux drm-lease- 
v3 drm-sequence-64-v2 


e git://people.freedesktop.org/~keithp/xserver drm-lease 


e git://oeople.freedesktop.org/~keithp/randrproto drm- 
lease 


e git://people.freedesktop.org/~keithp/xcb/proto drm- 
lease 


e git://people.freedesktop.org/~keithp/drm drm-lease 


e git://people.freedesktop.org/~keithp/mesa drm-lease 
drm-lease-intel 
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